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A  SCALABLE  SOFTWARE  PLAN  FOR  THE  UNDERSEA  WARHEADS’ 
MODELING  AND  SIMULATION  PROGRAM 


EXECUTIVE  SUMMARY 

The  research  plan  entails  the  development  and  implementation  of  a  scalable,  fully  coupled  fluid- 
structure  interaction  (FSI)  model  for  simulating  the  response  of  geometrically  complex  structures 
subjected  to  unsteady  blast  and  bubble  loads  stemming  from  explosions  in  the  deep  ocean  and  the 
surf  zone  (partially  and  fully  saturated  sand).  This  multilevel,  multiyear  plan  will  couple  a  scalable 
finite  difference  hydrodynamics  code  for  modeling  the  generation  and  evolution  of  propagating 
explosive  blast  waves  in  a  medium  (water,  partially  or  fully  saturated  sand,  "dry"  soils  and  sands)  to  a 
scalable  finite  element  structural  mechanics  code  for  modeling  the  response  of  complex, 
multidimensional  structures  to  the  loads  and  stresses  stemming  from  the  medium.  This  coupled 
scalable  model  will  be  used  to  address  problems  of  vital  interest  to  the  Department  of  Defense  (DoD). 
The  ultimate  goal  of  this  plan  is  to  develop  software  capable  of  interfacing  several  scalable 
computational  fluid  dynamics  (CFD)  models  with  several  scalable  computational  structural  mechanics 
(CSM)  models.  As  a  precursor  to  this  ultimate  goal,  the  scalable,  multiphase,  FAST3D  CFD  code  will 
be  coupled  to  the  scalable,  LS-DYNA3D  Computational  Structural  Mechanics  (CSM)  code  using  the 
Dynamic  Virtual  Cell  Embedding  (DVCE)  technique  as  the  interface. 

This  plan  will  couple  the  ongoing  research  and  parallel  development  efforts  among  Navy,  Army, 
and  private  industry  scientists  and  engineers  conducting  large-scale  hydrodynamics  and  structural 
mechanics  simulations.  This  collaborative  team  will  exploit  existing  technologies  from  both  the 
hydrodynamics  and  structural  mechanics  arenas  and  will  develop,  validate,  document,  and 
demonstrate  the  scalable  algorithms  on  fluid-structure  interaction  problems  of  vital  interest  to  the 
Department  of  Defense  (DoD). 

The  implementation  of  the  fully  coupled,  scalable  CFD-CSM  model  will  result  in  two,  somewhat 
overlapping,  research  thrusts: 

(1)  a  capability  to  compute  unsteady  explosive  blast  and  bubble  loads  on  submerged  and 
surface  vessels  and  the  large  deformation  response  to  those  loads;  and 

(2)  an  operational  simulation  capability  to  assess  explosion  and  shock  propagation  effects  in  the 
surf  zone  (partially  saturated  sand/sediment  environments). 

In  addition,  the  plan  has  a  built-in  flexibility  that  will  allow  for  the  ability  to  compute  unsteady  blast 
and  shock  loads  on  complex  structures  fully  or  partially  buried  in  varying  geologies  and  the 
structural  response  to  those  loads.  External  loads  on  a  structure  surrounded  by  a  medium  other  than 
air  are  significantly  affected  by  the  response  of  the  medium,  and  the  resulting  structural  behavior  has 
never  before  been  accurately  predicted  because  of  inadequacies  in  fully  coupled  media-structure 
interaction  techniques.  The  benchmark  calculations  planned  for  this  program  will  address  problems 
of  critical  interest  to  the  DoD  that  are  not  amenable  to  solution  on  current  vector  platforms.  These 
demonstration  calculations  are  designed  to  improve  the  effectiveness  of  weapons,  enhance  the 
survivability  of  undersea  and  surface  platforms  against  blast  and  bubble  loads,  improve  the  predicted 
performance  of  mine  countermeasure  efforts  in  the  surf  zone,  and  improve  the  survivability/lethality 
of  geometrically  complex  structures  buried  in  varying  geologies. 


E-l 


A  SCALABLE  SOFTWARE  PLAN  FOR  THE  UNDERSEA  WARHEADS’ 
MODELING  AND  SIMULATION  PROGRAM 


GOALS  AND  OBJECTIVES 

By  using  this  software,  the  goals  are  to  develop  scalable,  computational  simulation  technology 
that  will  improve  the  effectiveness  of  weapons,  enhance  the  survivability  of  undersea  and  surface 
platforms  against  blast  and  bubble  loads,  and  enhance  the  performance  prediction  of  mine 
countermeasure  efforts  in  the  surf  zone.  In  addition,  the  flexibility  of  this  scalable  software  plan  will 
allow  for  a  research  effort  focused  on  the  clarification  of  lethality  and  survivability  issues  related  to 
the  structures  buried  in  (dry)  sand  and  soil.  To  achieve  these  goals,  a  scalable  computational  fluid 
dynamics  (CFD)  model  will  be  coupled  to  a  scalable  computational  structural  mechanics  (CSM) 
model  to  provide  the  capability  to  compute  unsteady  blast  and  explosive  bubble  loading  on 
submerged  and  surface  vessels  and  the  large  deformation  response  to  those  loads;  to  develop  an 
operational  simulation  capability  to  assess  explosion  and  shock  propagation  effects  in  coastal,  shallow 
water  (including  partially  saturated  sand/sediment  environments);  and  to  compute  unsteady  blast  and 
shock-loading  on  structures  buried  in  sand  or  soil  and  the  structural  response  to  those  loads. 

The  Department  of  Defense  (DoD)  has  undertaken  an  underwater  explosion  (UndEx)  project  to 
couple  underwater  explosion  hydrodynamics  with  new  capabilities  in  large  deformation  structural 
mechanics.  In  a  related  effort,  DoD  has  recognized  the  urgent  need  of  a  self-consistent  numerical 
capability  for  modeling  explosion  and  shock  effects  in  surf  zone  seafloor  sediments — primarily 
sands — with  seawater  and  free-air  pore  fluid,  to  accelerate  the  progress  of  performance  prediction  of 
mine  countermeasure  efforts  (MCM).  Modeling  the  response  of  geometrically  complex  structures 
buried  in  sand  or  soil  under  blast  and  shock  loads  is  comparable  to  the  MCM  problem  where  the  pore 
water  would  play  a  minor  role.  All  three  projects  are  similar  in  the  sense  that  they  involve  coupling 
an  accurate,  high-resolution  hydrodynamic  model  with  an  efficient,  large  deformation  computational 
structural  mechanics  model.  The  MCM  and  soil-structure  efforts  are  further  complicated  by  the  need 
to  incorporate  a  physics-based  constitutive  model  for  the  complex  medium  and  the  associated  stress 
terms  in  the  conservation  equations  for  the  medium. 

DESCRIPTION  OF  SCALABLE  SOFTWARE  PROJECT 

This  plan  couples  the  efforts  of  the  Naval  Research  Laboratory  (NRL),  Livermore  Software 
Technology  Corporation  (LSTC),  Naval  Surface  Warfare  Center  (NSWC),  and  the  U.S.  Army 
Engineer  Waterways  Experiment  Station  (CEWES).  The  NRL  contribution  to  this  effort  exploits  our 
expertise  in  complex-geometry  CFD  based  on  high-resolution,  flux-corrected  transport  (FCT) 
algorithms,  new  parallel  algorithms  for  simulating  dynamic  air-water  interfaces,  a  dynamic  version  of 
the  virtual  cell  embedding  technique  (DVCE)  that  will  serve  as  the  coupler  between  the 
hydrodynamics  code  (FAST3D)  and  structural  mechanics  code  (LS-DYNA3D)  and  an  elastic-plastic 
effective  stress  multimaterial  hydrodynamic  model  for  simulations  of  dry  sands  and  soils  and  the 
gassy  sediments  in  the  surf  zone  environment.  The  LSTC  contribution  is  focused  on  the 
parallelization,  optimization,  and  coupling  of  LS-DYNA3D,  a  general  purpose,  explicit,  three- 
dimensional  (3-D),  finite-element  application  used  to  analyze  the  large  deformation  response  of 
inelastic  solids  and  structures.  NSWC  has  many  years  of  experience  with  both  the  experimental  and 
modeling  aspects  of  the  fluid-structure  interaction  problem.  CEWES  has  extensive  experience  in 
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modeling  and  simulating  the  dynamic,  nonlinear  response  of  civilian  and  military  structures  to  blast 
loads  and  in  the  constitutive  modeling  of  various  geological  materials. 

The  parallel  FAST3D  CFD  code  with  the  virtual  cell  embedding  (VCE)  grid  generator  for 
complex  geometry  uses  a  globally  structured  grid.  FAST3D  is  a  compressible  fluid  dynamics  flow 
solver  applicable  to  problems  such  as  noise  generation  in  supersonic  jets,  reactive  flow  in  ram 
accelerators,  and  low-speed  flows  such  as  the  unsteady  airwake  over  a  complex  Navy  ship 
superstructure.  The  multiphase  version  of  FAST3D  is  being  used  to  model  wave-breaking  over 
complex  geometries  and  underwater  explosion  and  bubble  dynamics.  FAST3D  was  designed  to  be 
scalable  since  these  applications  require  large  grid  sizes  as  well  as  long  run  times  to  collect  unsteady 
flow  field  data.  FAST3D  solves  the  unsteady  3-D  transport  equations  using  the  FCT  algorithm  with 
the  VCE  method  for  complex  geometies.  The  FCT  algorithm  is  a  high-order,  monotone,  conservative, 
positivity  preserving  algorithm.  In  our  FAST3D  model,  this  version  of  the  FCT  algorithm  solves  the 
3-D  flow  equations  using  a  direction-split,  time-split  method,  essentially  creating  a  series  of  one¬ 
dimensional  (1-D)  equations  that  the  FCT  algorithm  time  integrates.  In  addition,  a  structured, 
orthogonal,  rectilinear  mesh  is  used  for  3-D  problems.  In  turn,  this  would  require  extremely  fine 
grids  for  accurate  solution  around  complex  bodies;  however,  the  VCE  method  circumvents  this 
problem. 

The  VCE  method  improves  the  accuracy  around  complex  bodies  without  sacrificing  speed  or 
memory.  While  the  VCE  method  uses  a  structured,  rectilinear  grid,  the  cells  with  the  VCE  method 
may  be  full  outside  the  body,  fully  inside  the  body,  or  partially  inside  the  body.  It  is  the  partially 
obstructed  cells,  those  cells  “cut”  by  the  surface  of  the  body,  that  are  given  special  treatment.  To 
correctly  compute  the  fluxes  in  these  partially  obstructed  cells,  the  unobstructed  face  areas  and 
volumes  are  needed.  These  cells  are  subdivided  into  a  number  of  smaller  subcells  with  which  the  areas 
and  volumes  can  be  calculated  to  arbitrary  accuracy.  Using  the  VCE  method,  only  those  cells  next  to 
the  body  are  subdivided.  The  term  “virtual”  is  used  since  the  subcells  embedded  within  a  cell  are 
not  stored  in  memory  and  therefore  are  not  integrated  in  the  flow  solution.  Central  processing  unit 
(CPU)  time  is  not  sacrificed  appreciably  since  only  those  cells  next  to  the  body  require  special 
treatment.  The  information  computed  from  the  subcells  (computed  once  and  stored)  is  stored  in  a  list 
at  the  location  indicated  by  a  pointer  stored  in  the  parent  cell.  The  division  into  subcells  can  be  made 
so  fine  that  the  body  is  essentially  smooth  without  staircasing.  VCE  has  been  shown  to  provide 
accurate  results  when  tested  on  a  series  of  2-  and  3-D  problems.  A  dynamic  version  of  VCE  has  been 
developed,  which  models  moving  body  boundaries  (motion  prescribed)  with  a  high  degree  of 
accuracy.  A  deformable  body  boundary  model  is  undergoing  development.  Obviously  DVCE 
requires  that  the  areas  and  volumes  be  recalculated  at  every,  or  nearly  every,  time  step.  This  is 
tantamount  to  developing  a  scalable  version  of  the  grid  initializer  that  is  under  development.  It  is  our 
goal  to  implement  DVCE  as  the  core  of  the  interface  algorithm  coupling  the  CFD  and  the  CSM 
models. 

LSTC  is  the  leading  developer  of  the  software  used  for  the  analysis  of  nonlinear  mechanics. 
LSDYNA3D  is  a  general-purpose,  explicit,  3-D  finite  element  application  used  to  analyze  the  large 
deformation  response  of  inelastic  solids  and  structures.  It  has  a  full  interactive  graphic  capability, 
which  enables  the  user  to  monitor  complex  simulations.  Because  of  its  design  as  a  general-purpose 
program,  versatility,  and  relative  ease  of  use  LS-DYNA3D  has  become  a  standard  in  a  number  of 
application  areas  and  industries  that  include  automotive,  aerospace,  metal  manufacturing,  plastics,  and 
defense.  The  results  generated  by  the  program  Lave  been  repeatedly  correlated  with  experimental 
data,  thus  providing  confidence  in  computer  simulations. 


A  Scalable  Software  Plan  for  the  Undersea  Warheads 


3 


The  program  can  be  used  on  a  wide  range  of  computer  architectures  from  workstations  to 
supercomputers.  A  massively  parallel  version  of  the  program  has  been  developed  recently  to  take 
advantage  of  the  new  hardware  platforms  that  will  most  likely  enable  the  solution  of  problems  that 
currently  cannot  be  run  because  of  insufficient  computing  resources.  Several  customers  are  currently 
using  the  beta  version  of  LS-DYNA3D  reprogrammed  to  run  on  massively  parallel  systems,  including 
the  CRAY  T3D,  Intel  Paragon,  and  the  IBM  SP2. 

NSWC  has  been  involved  in  both  the  experimental  and  modeling  and  simulation  aspects  of  the 
underwater  explosion  fluid-structure  interaction  problem  for  a  number  of  years.  The  Center  has 
extensive  experience  in  modeling  explosives,  and  there  is  a  significant  amount  of  experimental  data 
available  with  which  modeling  comparisons  can  be  made.  Also,  for  a  number  of  years,  engineers  have 
been  applying  an  Eulerian-Lagrangian  fluid-structure  interaction  model  (DYSMAS,  developed  by 
IABG,  Germany)  to  the  study  of  this  problem. 

TECHNICAL  IMPLEMENTATION  PLAN 

The  scalable  software  plan  entails  the  development  of  a  scalable,  fully  coupled  structure-medium 
interaction  model  for  the  simulation  of  large  deformation  structural  response  to  unsteady  blast  and 
bubble-loading  stemming  from  underwater  explosions  in  both  deep  ocean  and  surf  zone 
environments.  This  multilevel,  multiyear  plan  will  couple  a  scalable  finite  difference  hydrodynamics 
model  (FAST3D)  with  a  scalable  finite  element  structural  mechanics  model  (LS-DYNA3D)  to  address 
problems  of  vital  interest  to  the  DoD.  The  program  for  the  outyears  of  this  endeavor  will  be 
generalized  to  incorporate  other  hydrodynamic  and  structural  mechanics  models  that  have  extended 
to  parallel  architectures. 

The  intrinsically  parallel  nature  of  the  FAST3D  algorithms  and  data  structure  ensures  that  the 
multimaterial  version  of  the  code  can  be  ported  with  little  difficulty  to  current  state-of-the-art 
computers  and  parallel  computers  with  different  architectures.  The  risk  for  the  scalable 
implementation  of  the  multimaterial  FAST3D  hydrodynamics  model  is  very  low,  as  the  existing  code 
requires  only  refinement.  This  aspect  of  the  development  will  be  funded  through  other  sources.  The 
vector  version  of  the  multimaterial,  elastic-plastic  hydrodynamic  effective  stress  model  follows  the 
same  structure  as  FAST3D  and,  as  such,  the  risk  for  the  scalable  implementation  of  this  algorithm  is 
low. 


A  parallel  version  of  LS-DYNA3D  has  been  implemented  by  LSTC.  It  requires  effort  to  optimize 
domain  decomposition  and  improve  the  efficiency  of  the  contact  search  algorithms.  The  risk  element 
in  this  endeavor  is  low 

Implementation  of  a  scalable  version  of  DVCE  entails  parallelization  of  the  grid  initializer.  Only 
those  cells  next  to  the  body  require  special  treatment,  and  the  information  computed  from  the 
subcells  is  stored  in  a  list  at  the  location  indicated  by  a  pointer  stored  in  the  parent  cell.  If  each 
processor  has  sufficient  memory  or  if  the  body  geometry  is  analytic  (cylinders,  spheres,  etc),  this  task 
is  trivial.  With  extremely  complex  and  detailed  geometries,  the  effort  may  involve  judicious  domain 
decomposition.  Overall,  the  risk  is  small.  Developing  the  coupler  itself  between  the  CFD  module  and 
the  CSM  module  does  involve  some  risk.  We  intend  to  minimize  the  risk  by  first  coupling  the  vector 
version  of  FAST3D  to  LS-DYNA3D  with  DVCE  as  the  interface  using  a  message-passing  protocol 
(e.g.,  MPI  or  PVM).  This  approach  will  also  enhance  cross-platform  portability  and  software 
reusability.  Development  and  verification  of  the  vector  form  of  the  coupler  v/ill  be  funded  by  other 
sources.  The  risk  in  this  aspect  of  the  initiative  is  moderate. 
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TARGET  COMPUTER  SYSTEMS  AND  REQUIRED  RESOURCES 

The  intent  is  to  develop  scalable  software  that  is  portable  on  a  wide  range  of  systems.  Specific 
systems  include  Cray  T3D,  IBM  SP-2,  Intel  PARAGON,  and  clustered  workstations.  For  production, 
each  individual  simulation  may  require  up  to  several  hundred  Cray  C-90  equivalent  hours.  The 
3-year  estimated  hours  would  be  approximately  8000  (C-90)  equivalent  hours. 

There  is  a  wide  range  of  both  vector  and  parallel  platforms  available  to  the  agencies  of  the  DoD. 
Typically,  requests  for  proposals  for  computer  allocations  on  the  High  Performance  Computing 
Modernization  Plan  (HPCMP)  systems  are  released  in  August.  Computer  time  on  the  HPCMP  systems 
are  allocated  by  using  a  proposal  submission  and  review  process.  Typically,  there  is  a  single  call  for 
proposals  for  the  allocations  on  the  HPCMP  systems  currently  available;  subsequently  calls  for 
allocations  on  new  systems  may  be  submitted  when  they  become  available.  The  Appendix  contains  a 
summary  listing  of  the  resources  currently  available. 

DEMONSTRATION  CALCULATIONS  AND  RELEVANCE 
TO  CRITICAL  DOD  CSM  PROBLEMS 

The  demonstration  calculations  will  directly  address  critical  DoD  structure-medium  interaction 
problems  associated  with  weapons  effectiveness,  enhanced  lethality  and  survivability  of  undersea  and 
surface  platforms,  improved  performance  prediction  of  mine  countermeasure  efforts,  and  improved 
survivability  and  lethality  of  buried  structures.  The  latter  effort  is  included  in  this  plan  solely  for 
completeness.  Extensive  benchmark  tests  range  from  shock  and  bubble-loading  on  single-  and 
double-walled  cylinders  to  shock  and  bubble-loading  on  double-walled  submarines,  explosive  shock¬ 
loading  of  fully  or  partially  buried  mines  in  the  surf  zone  and  of  structures  buried  in  sand  or  soil. 
There  is  an  extensive  database  available  to  make  detailed  comparisons  with  experiments  to  ensure 
rapid  progress  and  verification. 

The  development  and  implementation  of  a  scalable,  fully  coupled  structure-medium  interaction 
model  has  significant  dual-use  applications.  There  is  the  potential  for  improved  hull  design  for 
commercial  vessels  displaying  reduced  susceptibility  to  wave  damage  and  reduction  of  cavitation- 
induced  shafting  and  machinery  vibration,  double-hulled  tanker  design,  earthquake-safe  double¬ 
hulled  storage  tanks,  complex  geometry  oceanic  drag,  predictions  for  harbor  and  shore-line 
maintenance,  pier  damage  caused  by  wave  action  and  storms,  and  prediction  of  the  evolution  of  the 
bethnic  boundary  layer  in  response  to  storms,  dredging,  and  other  man-made  disturbances,  to  cite  just 
a  few. 

SCALABLE  SOFTWARE  PRODUCTS  &  OTHER  DELIVERABLES 

Project  deliverables  and  participants  are  listed  below  by  year.  Additonal  deliverables  include 
refereed  publications,  progress  reports,  conference  proceedings,  and  benchmark  comparisons  and 
documentations.  The  agencies  involved  in  completing  the  tasks  are  also  listed. 

FIRST  YEAR 

•  Implement  scalable  version  of  the  multimaterial  FAST3D  hydrodynamic  model  (NRL); 
includes  developing  scalable  versions  of  reactive  burn  models  and  tabular  EoS. 


Develop  scalable  version  of  the  DVCE  coupler  (NRL). 
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•  Begin  optimization  of  scalable  version  of  the  LS-DYNA3D  structural  mechanics  model 
(LSTC);  includes  improving  efficiency  of  contact  search  algorithm  and  modifying/optimizing 
domain  decomposition  for  the  noncontact  case. 

•  Begin  implementation  of  automated  domain  decomposition  for  the  contact  case  (LSTC). 

•  Couple  vector  version  of  FAST3D  to  LS-DYNA3D  with  DVCE  using  a  message  passing 
protocol  (e.g.,  MPI,  PVM)  (NRL,  LSTC,  CEWES,  NSWC). 

•  Benchmark  calculations  against  theoretical  and  experimental  results. 

•  Benchmark  all  scalable  models  against  vector  versions  of  the  same  models  (NRL,  LSTC, 
CEWES,  NSWC). 

SECOND  YEAR 

•  Couple  scalable  version  of  FAST3D  to  scalable  version  of  LS-DYNA3D  using  DVCE  (NRL, 
LSTC);  includes  improving  efficiency  of  locating  the  wet  surface. 

•  Incorporate  scalable  model  of  3-D  stress  terms  into  FAST3D  hydrodynamic  model  (NRL, 
CEWES). 

•  Optimize  LS-DYNA3D  to  take  advantage  of  efficient  load  balancing  (LSTC). 

•  Complete  benchmark  tests  for  shock/explosion  bubble  loading  on  single-  and  double-walled 
cylinders  and  single  walled  spheres  (NRL,  LSTC,  NSWC);  including  end  closures  on  cylinders 
with  and  without  internal  supports. 

•  Begin  benchmark  tests  for  shock/explosion  bubble  loading  on  single-hulled  ships  and 
submarines  (NRL,  LSTC,  NSWC). 

•  Begin  benchmark  tests  for  explosive  loading  of  structure  buried  in  dry  soil  (NRL,  LSTC, 
CEWES). 

THIRD  YEAR 

•  Enhance  completed  scalable  FSI  model  to  be  portable  on  a  wide  range  of  systems  (NRL, 
LSTC). 

•  Complete  above  benchmark  tests,  if  not  yet  completed  (NRL,  LSTC,  NSWC). 

•  Perform  additional  benchmark  tests  from  among: 

shock/explosion  bubble  loading  on  a  full  ship  hull; 

shock/explosion  bubble  loading  on  double-hulled  submarine; 

explosive  shock  loading  on  fully  or  partially  buried  mine  in  the  surf  zone;  and 

explosive  shock  loading  on  structure  buried  in  sand/soil  (NRL,  LSTC,  NSWC,  CEWES). 

•  Generalize  structure-medium  coupler  to  incorporate  other  hydrodynamic  and  structural 
models  (NRL,  LSTC,  CEWES,  NSWC). 
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SUMMARY 

The  goal  of  this  scalable  software  plan  is  to  develop  and  implement  a  scalable,  fully  coupled 
fluid-structure  interaction  (FSI).  model  for  simulating  the  response  of  geometrically  complex 
structures  subjected  to  unsteady  blast  and  bubble  loads  stemming  from  explosions  in  deep  ocean  and 
the  surf  zone  (partially  and  fully  saturated  sand).  This  multilevel,  multiyear  plan  will  couple  a 
scalable,  finite  difference  hydrodynamics  code  for  modeling  the  generation  and  evolution  of 
propagating  explosive  blast  waves  in  a  medium — water,  partially  or  fully  saturated  sand,  "dry"  soils 
and  sands — to  a  scalable,  finite  element  structural  mechanics  code  for  modeling  the  response  of 
complex,  multidimensional  structures  to  the  loads  and  stresses  stemming  from  the  medium.  This 
coupled  scalable  model  will  be  used  to  address  problems  of  vital  interest  to  the  DoD.  The  ultimate 
goal  of  this  plan  is  to  develop  scalable  software  that  would  be  capable  of  interfacing  several  scalable 
computational  fluid  dynamics  (CFD)  models  with  several  scalable  computational  structural  mechanics 
(CSM)  models.  As  a  precursor  to  this  ultimate  goal,  the  scalable,  multiphase  FAST3D  Computational 
Fluid  Mechanics  (CFD)  code  will  be  coupled  to  the  scalable  LS-DYNA3D  Computational  Structural 
Mechanics  (CSM)  code  using  the  Dynamic  Virtual  Cell  Embedding  (DVCE)  technique  as  the 
interface. 

Two  parallel  computing  architectures  are  currently  popular:  single-instruction,  multiple  data 
(SIMD)  and  multiple  instruction  multiple-data  (MIMD).  In  the  SIMD  approach,  processors  perform 
their  operations  in  unison  under  the  control  of  a  master  processor.  SIMD  machines  operate 
efficiently  on  nearest-neighbor  transfer  of  information;  however,  general  data  exchange  operations, 
as  required  for  unstructured  grids,  are  very  inefficient.  Special  routers  or  precompilers  have  been 
developed  in  an  attempt  to  circumvent  this  problem.  These  machines  are  not  practical  for  more 
general  types  of  applications  where  the  grid  topology  changes  every  few  time  steps  (remeshing, 
h-refinement,  etc.).  Examples  of  SIMD  architecture  with  distributed  memory  are  the  Connection 
Machines  CM-1,  CM-2,  and  CM-200,  and  the  MASSPAR-1. 

For  MIMD,  processors  operate  independently  on  their  data,  possibly  using  different  instructions, 
with  coordination  between  the  processors  handled  by  the  explicit  passing  of  messages  and 
synchronization  hardware.  Most  of  the  algorithmic  considerations  designed  for  vector  machines  are 
applicable  to  many  MIMD  machines,  since,  in  many  current  designs,  each  node  of  an  MIMD  system 
is  a  vector  processor.  The  software  development  for  MIMD  machines  is,  in  many  instances,  more 
complex  than  that  for  SIMD.  The  Intel  series  of  machines — the  Cray  T3D,  IBM  SP1  and  SP2,  and  the 
CM-5 — are  examples  of  MIMD  architecture.  In  general,  MIMD  machines  provide  a  more  flexible 
computing  environment  because  different  processors  can  work  simultaneously  on  different  subtasks. 
Overall,  a  MIMD  computer  is  a  more  general  purpose  computer  than  a  SIMD  computer. 

Since  the  major  component  of  this  effort  is  to  merge  two  diverse  software  packages,  significant 
time  and  effort  must  be  expended  in  coordinating  the  data  structures  of  the  two  codes.  This  will 
prove  to  be  especially  beneficial  when  the  basic  structure  is  generalized  to  incorporate  other 
hydrodynamic  codes  or  structural  dynamics  codes.  The  data  structure  should  be  designed  in  such  a 
manner  as  to  facilitate  check-point  restarting,  provide  fault  tolerance  for  very  long  run  times,  allow 
for  changes  in  the  number  of  physical  nodes  used  in  the  different  segments  of  a  single  simulation, 
and  ensure  that  dump-restart  files  are  compatible  with  other  serial  processors  required  for  initializing 
complex  geometry  cases  and  postprocessing  the  simulation  results.  In  addition,  portability  of  the 
individual  and  complete  software  packages  must  be  ensured.  The  final  software  package  should  be 
flexible  enough  to  run  on  several  different  architectures  with  minimal  changes. 
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PROJECT  MANAGEMENT  PLAN 

The  Naval  Research  Laboratory’s  Laboratory  for  Computational  Physics  and  Fluid  Dynamics, 
the  Livermore  Software  Technology  Corporation,  the  U.S.  Army  Corp  of  Engineers,  Waterways 
Experiment  Station,  and  the  Naval  Surface  Warfare  Center,  White  Oak,  will  serve  as  project  managers 
to  assure  successful  completion  of  this  effort.  There  is  a  history  of  joint  technological  efforts  between 
NRL  and  NSWC  and  between  LSTC  and  NSWC.  Frequent  dialog  among  the  software  developers  at 
their  respective  headquarters  is  planned  and  required  in  order  to  ensure  progress.  Project  reviews  and 
annual  reports  will  document  the  progress  outlined  above. 
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Appendix 


DoD  High-Performance  Computing 
Modernization  Plan  Resources 


System  Configuration 

Site 

Availability 

Service  Unit  (SU) 

Definition 

Cray  C90,  16  processors,  1  GW  memory, 

4  GW  SSD  plus  300  GB  disk  and  50  TB  archival  storage 

Navy,  NAVO 

Stennis  Space  Ctr.,  MS 

Now 

1  CPU  hour  = 

3  SU’s 

Cray  Y-MP,  8  processors,  128  MW  memory,  and  256 
MW  SSD,  120  GB  disk,  and  25  TB  archival  stroage 

Navy,  NAVO 

Stennis  Space  Ctr.,  MS 

Now 

Classified 

1  CPU  hour  = 

1  SU 

Cray  C90,  16  processors,  1024  MW  memory,  2048 

MW  SSD,  256  GB  disk,  and  access  to  10  TB  robotic 
archival  storage 

Army,  CEWES 

Vicksburg,  MS 

Now 

1  CPU  hour  = 

3  SU’s 

Cray  Y-MP,  8  processors,  128  MW  memory, 

256  MW  SSD,  250  GB  disk,  and  access  to  10  TB 
archival  storage 

Army,  CEWES 
Vicksburg,  MS 

Now 

1  CPU  hour  = 

1SU 

SGI  Power  Challenge,  96  processors,  16  GB  memory, 

80  GB  disk  storage 

Army,  ARL 

Aberdeen,  MD 

Now 

1  CPU  hour  on 

1  processor 

node  =  1  SU 

Cray  2,  4  processor,  256  MW  memory,  100  GB  disk 
storage 

Army,  ARL 

Aberdeen,  MD 

Now 

40%  of  system 

available  to 

HPCMP 

Classified 

1  CPU  hour  = 

1  SU 

Intel  Paragon  XP/S,  352  processors,  32  MB  memory 
per  processor,  44  GB  disk  storage  plus  22  GB 
workspace 

Air  Force,  WPASC 
Dayton,  OH 

Now 

1  wall  clock  hour  on 

1  processor 

node  =  1  SU 

CM-5E,  256  processor  s,  32  GB  memory  150  GB  SDA, 
150  GB  disk,  6  TB  archival  storage 

Navy,  NRL 

Washington,  DC 

Now 

40%  of  system 

available  to 

HPCMP 

1  CPU  hour  on 

32  processor 

nodes  =  1  SU 

CM-5,  896  processors,  28  GB  memory 

Army,  AHPCRC 
Minneapolis,  MN 

Now 

50%  of  system 

available  to 

HPCMP 

1  CPU  hour  on 

32  processor 

nodes  =  1  SU 

IBM  SP2,  400  nodes,  64-1024  MB  memory  per  node, 
1-4.5  GB  disk  per  node,  5  TB  archival  storage 

Air  Force,  MHPCC 

Maui,  HI 

Now 

50%  of  system 

available  to 

HPCMP 

1  CPU  hour  on 

1  processor 

node  =  1  SU 
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IBM  SP2,  80  nodes,  20  GFLOPS 

Air  Force,  MHPCC 

Maui,  HI 

Now 

50%  of  system 
available  to 

HPCMP 

Classified 

1  CPU  hour  on 

1  processor 

node  =  1  SU 

Cray  T3D,  128  nodes,  8  GB  memory,  140  GB  disk 
storage,  1.2  TB  archival  storage 

Air  Force,  EGLIN 

Eglin  AFB,  FL 

Now 

1  wall  clock  hour  on 

1  processor 

node  =  1  SU 

Convex  Exemplar,  32  processor,  8  GB  memory, 

160  GB  disk 

Navy,  NCCOSC 

San  Diego,  CA 

Now 

1  wall  clock  hour  on 

1  processor 

node  =  1  SU 

Intel  Paragon  XP/S,  336  processors,  304  nodes  with 

32  MB  per  node,  32  nodes  with  128  MB  per  node,  130 
GB  disk,  plus  65  GB  archival  storage 

Navy,  NCCOSC 

San  Diego,  CA 

Now 

Classified 

1  wall  clock  hour  on 

1  processor 

node  =  1  SU 

Additional  system  upgrades  and  new  systems  (to  be  determined)  will  be  announced  as  available. 


